Relevance of the Heisenberg-Kitaev model for the honeycomb lattice iridates ^4 2 Ir03 
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Combining thermodynamic measurements with theoretical density functional and thermodynamic calcula- 
tions we demonstrate that the honeycomb lattice iridates A2lr03 (A =Na, Li) are magnetically ordered Mott 
insulators where the magnetism of the effective spin-orbital S=l/2 moments can be captured by a Heisenberg- 
Kitaev (HK) model with Heisenberg interactions beyond nearest-neighbor exchange. Experimentally, we ob- 
serve an increase of the Curie-Weiss temperature from « —125 K for Na2lr03 to « —33 K for Li2lr03, 
while the antiferromagnetic ordering temperature remains roughly the same Tn ~ 15 K for both materials. 
Using finite-temperature functional renormalization group calculations we show that this evolution of 0, Tn, 
the frustration parameter / = 6/Tn, and the zig-zag magnetic ordering structure suggested for both materials 
by density functional theory can be captured within this extended HK model. Combining our experimental 
and theoretical results, we estimate that Na2lr03 is deep in the magnetically ordered regime of the HK model 
(a « 0.25), while Li2lr03 appears to be close to a spin-liquid regime (0.6 < a < 0.7). 



Introduction- The fundamental importance of the Ki- 
taev model, which describes the highly anisotropic exchange 
of SU(2) spin- 1/2 moments on the honeycomb lattice, has 
quickly been appreciated for its rare combination of micro- 
scopic simplicity and an exact analytical solution [1]. It has 
also become an archetypal example of a microscopic model 
that - depending on the spatial anisotropy of its couplings 
- harbors three of the currently most sought-after collective 
states in condensed matter physics \2]: a gapless spin liq- 
uid with emergent Majorana fermion excitations, a gapped 
Z2 spin liquid, and a topologically ordered phase with non- 
Abelian quasiparticle statistics (in the presence of a magnetic 
field perpendicular to the honeycomb lattice) JH Espe- 
cially, physical realizations of topological states of the latter 
form which support Majorana fermion zero modes [4] are in- 
tensely searched for in various candidate systems including 
certain fractional quantum Hall systems [5], unconventional 
superconductors [6], as well as heterostructures of topological 
insulators, semi-metals, or semiconductors with conventional 
s- wave superconductors Jz|-|9|], not only because of their fun- 
damentally new character but also due to their possible appli- 
cation in topological quantum computation proposals flXOh . 

A direct realization of the Kitaev model could provide yet 
another alternative path to this goal. First proposals to en- 
gineer implementations of the Kitaev model were discussed 
in the context of optical lattices 111 111 and superconducting 
circuits 11211 . More recently, it has been put forward that 
strong spin-orbit coupling in certain Mott insulating transition 
metal oxides IB, 14] could inherently give rise to Kitaev- 
type couplings of effective spin-orbital degrees of freedom. 
Among the best candidate materials are layered iridates of the 
form A2lrC>3, which exhibit Mott insulating ground states and 



where the Ir 4+ form effective S =1/2 moments as it was re- 
cently observed for Na2lrC>3 [15]. On a microscopic level, it 
has been argued that the strong spin-orbit coupling in these 5d 
transition metal systems leads to orbital dependent anisotropic 
in-plane exchange that precisely mimics the Kitaev couplings 
II 1 311 . For real materials, however, further interactions will in- 
evitably be present and in particular one might expect that the 
original spin exchange is not completely damped and isotropic 
Heisenberg interactions will compete with the anisotropic Ki- 
taev couplings [14]. Such a Heisenberg-Kitaev (HK) model 
can be written down in its simplest form as 



#hk = (1 - a) , y]<Ji.<Tj 



(1) 



where the Gi are Pauli matrices for the effective S = 1/2 and 
7 = x,y,z labels the three different links for each spin of the 
honeycomb lattice. It has been shown [14] that the isotropic 
Heisenberg exchange in the first term of model © enters as 
antiferromagnetic coupling, while the anisotropic Kitaev ex- 
change (in the second term) is ferromagnetic. Varying the 
relative coupling a of the two exchange terms a sequence of 
three different phases has been found JH 0, [HI] : a conven- 
tional Neel antiferromagnet for < a < 0.4, a so-called a 
stripy antiferromagnet for 0.4 < a < 0.8, and a spin-liquid 
state for 0.8 < a < 1. 

But while the A2lr03 materials have been appreciated from 
a theory perspective as possible candidate materials to look 
for Kitaev-like and HK-like physics there has so 

far been very limited experimental data available for these 
layered iridates. For Li2lrC>3 there have been two conflict- 
ing reports llTi |T8|], with one report [17] suggesting param- 
agnetic behavior between T = 5 K and 300 K without any 
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sign of magnetic order, while the second report [18] indicated 
an anomaly in the magnetic susceptibility below T = 15 K, 
which was also accompanied by a hysteresis between zero- 
field-cooled and field-cooled data, suggesting glassy behavior. 
No heat capacity data has so far been available for Li2lr03. 

For single crystal Na2lr03 some of us have earlier shown 
Jl5ll Mott insulating behavior with antiferromagnetic order- 
ing below Tn = 15 K. Subsequent resonant magnetic x-ray 
scattering measurements Jl9ll on single crystals were consis- 
tent with either stripy or zig-zag magnetic order, with supple- 
mentary DFT calculations indicating that zig-zag order was 
the more likely magnetic ground state for Na2ir03. This 
gave rise to another theoretical puzzle, since the original HK- 
model with nearest-neighbor exchange, i.e. model ([TJ, allows 
for stripy magnetic ordering but not zig-zag order. Finite- 
temperature calculations ||6|] for model © pointed to another 
discrepancy with experimental observations, since the theoret- 
ical calculations indicated that the competition of the Heisen- 
berg and Kitaev exchanges in model 0} does not lead to a sub- 
stantial suppression of the magnetic ordering transition with 
regard to the Curie- Weiss scale and the frustration parameter 
/ = |0|/T/v was found to never exceed / w 2 [6], while for 
Na2ir03 experiments indicate / w 8 [15]. Pieces of this puz- 
zle were recently solved when it was shown that taking into 
account Heisenberg interactions beyond the nearest neighbor 
exchange can indeed stabilize the zig-zag ordering pattern 
J4l|2l|]. For antiferromagnetic exchanges, the latter are also 
expected to introduce geometric frustration. In the following 
we will expand this discussion of the role of further neighbor 
Heisenberg exchange and by providing a detailed comparison 
of theoretical and experimental results we will establish a mi- 
croscopic description of the layered iridates A2ir03 in terms 
of such an extended Heisenberg-Kitaev model. 

Quickly summarizing our main results we report magnetic 
and heat capacity measurements on high quality polycrys- 
talline samples of A2ir03 (A = Na, Li). Magnetic mea- 
surements show local moment behavior with effective spin 
S = 1/2 moments. Both magnetic and heat capacity measure- 
ments show sharp anomalies at Tn = 15 K for both materials 
indicating bulk antiferromagnetic ordering. For both materi- 
als our DFT calculations indicate that the most likely mag- 
netic order is of zig-zag type. Finite-temperature functional 
renormalization group (FRG) calculations for an extended HK 
model including next-nearest (J 2) and next- to-next-nearest 
( J3) neighbor antiferromagnetic Heisenberg interactions are 
then used to confirm the type of magnetic order, and study 
the evolution of the Curie- Weiss temperature scale 6, the or- 
dering scale Tn, and the frustration parameter / = \6\/Tn 
as a and the various J's are varied. It must be emphasized 
that in contrast to the classical phase diagram of the extended 
Heisenberg-Kitaev model discussed earlier! 4] our FRG calcu- 
lations are performed on the quantum level. We show that the 
experimentally observed evolution of 6, Tn, and / and the 
observed magnetic order can all be very well captured within 
this extended HK model. Comparison of experiments with 
calculations suggest that while the Kitaev term is small for 
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FIG. 1 : (Color online) Magnetic susceptibility % versus temperature 
T for A 2 lr0 3 (A = Na, Li). The fit by the Curie- Weiss (CW) expres- 
sion x — Xo + C/(T — 0) is shown as the curve through the data. 
The insets (a) and (b) shows the anomaly at the antiferromagnetic 
ordering for the Na and Li systems, respectively. 

Na2ir03, the Li2ir03 system with 0.6 < a < 0.7 sits quite 
close to the spin-liquid state in the Kitaev limit a > 0.8. 

Magnetic Susceptibility- The magnetic susceptibility 
X = M/H versus T data for A 2 lrOs (A = Na, Li) are shown 
in Fig. [[J The x(T) data between T = 150 K and 300 K 
were fit by the Curie- Weiss expression x = Xo + j&q with 
Xo, C, and 6 as fitting parameters. The fit, shown in Fig. [T] 
as the solid curve through the data, gives the values xo = 
3.6(4) x 10- 5 cm 3 /mol, C = 0.40(2) cm 3 K/mol, and = 
-125(6) K, forNa 2 Ir0 3 and, xo = 8.1(7) x 10~ 5 cm 3 /mol, 
C = 0.42(3) cm 3 K/mol, and = -33(3) K, for Li 2 Ir0 3 , re- 
spectively. Assuming a ^-factor g = 2 the above values of C 
correspond to an effective moment of /i e ff = 1.79(2) /ib and 
/i e ff = 1.83(5) /jlb, for Na2ir03 and Li2lr03, respectively. 
These values of /i e ff are close to the value 1.74 /iB expected 
for spin =1/2 moments. This local-moment formation along 
with the insulating resistivity (see auxiliary material [12 511 ) in- 
dicates that like its sister compound Li 2 Ir03 is indeed a Mott 
insulator. The value of the Weiss temperature 6 = — 33(3) K 
for Li2lrC>3 further suggests that the effective antiferromag- 
netic exchange interactions have weakened when compared 
to the Na2lr03 system. However, the xCO data for Li2ir03 
also show an anomaly at 15 K suggesting that an antiferro- 
magnetic transition occurs at roughly the same temperature as 
for Na2lr03. This is further supported by our heat capacity 
results presented below. The insets (a) and (b) in Fig. Q] show 
the x(^) data at low temperatures to highlight the anomaly 
seen at the onset of the antiferromagnetic transition below 
Tn = 15 K in both materials. 

Heat Capacity- In Fig. [21 we show heat capacity data di- 
vided by temperature C/T versus temperature T for A2lr03 
(A = Na, Li), and for the non-magnetic analog Li2SnC>3. The 
anomaly at T N = 15 K in the data for both A 2 Ir0 3 (A = Na, 
Li) materials confirms bulk magnetic ordering. A small bump 
is also observed around T = 5 K in the C(T) for Li 2 Ir03. 
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FIG. 2: (Color online) Heat capacity divided by temperature C/T 
versus T data between T = 1.8 K and 50 K for A 2 Ir0 3 (A =Na, Li) 
and the non-magnetic analog Li 2 Sn03. The anomaly at Tn = 15 K 
for both A2lr03 materials indicates onset of bulk antiferromagnetic 
order. The inset shows the difference heat capacity AC(T) and the 
difference entropy S(T) for Li2lr03. 



This most likely arises due to a small amount (< 5%) of dis- 
order in the sample. [3] The magnetic contribution AC(T) for 
Li2ir03 shown in the inset of Fig. [2] was obtained by subtract- 
ing the C(T) data of Li 2 Sn03 from the data of Li 2 Ir03. The 
latter reveals a clearly more visible lambda-like anomaly at 
Tn = 15 K. A slight depression of Tn in an applied mag- 
netic field of H = 9 T was observed (not shown) which 
points to the antiferromagnetic nature of the magnetic order- 
ing in Li2lrC>3. The entropy S(T) obtained by integrating the 
AC/T versus T data is also shown in Fig.|2]inset. Just above 
Tn the entropy is only about 15%Rln2. Such a reduced en- 
tropy at the transition was also observed earlier for single crys- 
talline Na9lrQq. lll5ll The small entropy points to the reduced 
ordered moment and the possible proximity to a non-magnetic 
ground state. 

Magnetic Ordering- From the similarities in the anoma- 
lies seen in \ an d C data for both the Na and Li systems, 
it would seem likely that the kind of magnetic order would 
also be similar for the two. To resolve the magnetic struc- 
ture for Li2lrC>3, spin density function calculations within 
the LDA+U+SO approximation were performed for the Neel, 
stripy, and zig-zag configurations with the moments con- 
strained along the crystallographic axes [Qjlllll]. The results 
are summarized in Table U We find that as for Na2lrC>3 111911 . 
the zig-zag configuration has the lowest energy and is hence 
the most likely magnetic structure for Li2lrC>3. 

We now turn to the evolution of magnetic properties as we 
go from the Na to the Li compound. From our x(T) data 
above we find that the Curie- Weiss temperature decreases 
from « —120 K to ~ —33 K on going from Na2lrC>3 to 
Li2lrC>3 possibly indicating that the effective magnetic inter- 
actions are weaker for Li2lrC>3. Surprisingly however, both 
x(T) and C{T) show that both materials order magnetically 
at roughly the same temperature T/v ~ 15 K. The frustra- 



tion parameter / = 0/Tn, however, reduces from « 8 for 
Na 2 Ir0 3 to ^ 2 for Li 2 Ir0 3 . 

In previous theoretical calculations 06D for the thermody- 
namics of the HK model Q]), the ordering temperature T/v was 
found to be largely insensitive to variations of a whereas 6 
was found to decrease monotonically with a within the stripy 
magnetic phase. The increase of the Weiss temperature scale 
on increasing a is a direct consequence of the fact that the two 
coupling terms in the HK model (Q]) enter with opposite cou- 
pling signs. Increasing the relative strength of the ferromag- 
netic Kitaev term thus leads to an increase of the Weiss tem- 
perature scale. These theoretical trends thus seem to match 
well with what is observed in our experiments. 

There are, however, two issues where experiments differ 
from predictions for the HK model. First, the zig-zag mag- 
netic order obtained from DFT for both Na2lr03 and Li2lrC>3 
is not one of the three phases of the HK model Il4i1 . Secondly, 
a maximum frustration parameter / « 2 was found in calcu- 
lations for the HK model[ 6], which is much smaller than the 
experimentally observed value / « 8 for Na9irOq |[l5ll . 

To resolve these discrepancies it has recently been argued 
||4|] that further nearest neighbor antiferromagnetic Heisen- 
berg exchange interactions should be added to the original 
HK model, which can indeed stabilize the zig-zag magnetic 
order. It was further demonstrated [|4|] that the experimental 
magnetic susceptibility data for ^IrOs (A = Na, Li) mate- 
rials can only be fit when including the Kitaev term in this 
expanded microscopic model. 

The inclusion of further than nearest-neighbor antiferro- 
magnetic interactions is further expected to introduce geomet- 
ric frustration in addition to the frustration arising from the 
competition between the Heisenberg and Kitaev couplings of 
the original model. We have therefore expanded our FRG cal- 
culations [6] to such an extended HK-J2,J3 model and de- 
termine its thermodynamic properties by extracting the high- 
temperature CW behavior (from the RG flow), the onset of 
magnetic ordering (from the breakdown of the RG flow), 
and the nature of the various ground states by calculating 
momentum-resolved magnetic susceptibility profiles as fur- 
ther detailed in the auxiliary material fl25ll . We focused our 
calculations on the parameter regime 0.2 < a < 0.8 and 
< J2, J3 < 1. A representative plot of the ordering scale 
A c as a function of a for fixed J2 = J3 = 0.6 is shown in 
Figure [3] with the inset showing the evolution of the frustra- 
tion parameter /. Our calculations indicate that zig-zag order 
is stabilized for an extended range 0.25 < a < 0.7 in agree- 
ment with recent calculations for the HK-J2, J3 model 0]. 
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TABLE I: Total energy E to t per Ir for Li 2 Ir03 for three anti- 
ferromagnetic configurations, obtained from collinearly constrained 
LDA+U+SO simulations. 



4 



0.25- 



< 

O 



0.2- 



0.15- 



0.1 



0.05- 



^3 

O go 



J 2 =J 3= 0.6 




zigzag 
spiral 
AFM 
stripy 





1 

l J 2 =J 3 =0.6 

\ J 2 =J 3 =0 











0.2 



0.4 0.6 

a 



0.8 



~n 1 1 1 1 — 

0.2 0.4 0.6 

coupling parameter 



0.25 



0.2 



0.15 



0.1 



0.05 



a 



FIG. 3: (Color online) Ordering scale A c obtained from the FRG 
calculations of the HK- J2 , J3 model as a function of a for J2 , J3 = 
0.6. The dashed line indicates the crossover between regions with 
magnetic order and regions with no long range order at both low and 
high a, respectively. A regime of enhanced numerical uncertainties 
is seen near a ^0.8. The inset shows the frustration parameter f as a 
function of a for J2, J 3 — 0.6 in comparison to the pure Heisenberg- 
Kitav model with J2, J 3 — 0. 



Around the Kitaev limit for a > 0.8 we find an extended 
non-magnetic spin-liquid phase, which connects directly to 
the one of the original HK model {]])• For a < 0.2 we 
obtain another non-magnetic ground state, evidently arising 
from the further nearest-neighbor exchange J 2 , J3 frustrating 
the nearest-neighbor J and suppressing the Neel state in favor 
of a valence bond dimer crystal 12111 . 

Our results summarized in Fig. further indicate that the 
potentially counter-intuitive experimental observation of T/v 
staying roughly the same in going from Na2lr03 to Li2lr03 
even though decreases by a factor of « 4 in fact agrees well 
with our calculations showing that the ordering scale stays 
more or less constant for the zig-zag ordered ground state in 
the regime 0.25 < a < 0.7. Finally, we note that our calcu- 
lations also indicate that the frustration parameter / = 0/Tn 
decreases monotonically with a in the region where magnetic 
order is found (see the inset in Fig. [3}, which is a direct conse- 
quence of the Curie- Weiss scale 6 decreasing monotonically 
in this region. Interestingly, for small a the geometric frus- 
tration induced by the further nearest-neighbor exchange be- 
comes more evident and the parameter / reaches values much 
larger than obtained for the original HK model [6] and in fact 
becomes comparable in size to what is observed experimen- 
tally for Na2lrC>3 where / w 8. 

To place the ^IrOs materials on the diagram in Fig. [3] we 
note that the zig-zag ordered ground state indicated in DFT 
calculations for both materials is found only in the parameter 
range 0.25 < a < 0.7 in the presence of significant sec- 
ond and third neighbor exchange. Additionally, an enhanced 
frustration parameter / is found only for small a > 0.25 be- 
fore the system transitions to a non-magnetic ground state (for 
a < 0.2) and the ordering temperature starts to drop drasti- 
cally. We therefore place Na2lr03 at a ~ 0.25. In contrast, 



Li2lrC>3 with zig-zag order at Tn = 15 K and / « 2 can be 
placed at a > 0.65 putting it considerably closer to the spin- 
liquid regime 11231] around the Kitaev limit beyond a > 0.8. 

Finally, we note that in going from the Na to the Li sys- 
tem the a, b lattice parameters are reduced by w 4.5% while 
the c parameter is reduced by w 10%. Thus, substituting Na 
by Li is equivalent to preferentially applying chemical pres- 
sure along the c axis (_L to the honeycomb planes). This most 
likely leads to the Ir06 octahedra becoming more symmetri- 
cal within the afr-plane which in turn enhances the parameters 
771,2 (in the notation of Ref.fl4h leading to an increased Kitaev 
coupling. This is consistent with the value of 0.6 < a < 0.8 
estimated above for Li2lr03 which puts its closer to the Kitaev 
limit. 

In summary, we have shown that magnetic properties of 
the Mott insulating iridates A2lr03, in particular the evolution 
of thermodynamic observables, i.e. the Curie- Weiss temper- 
ature 6 and ordering temperature T/v, as well as their low- 
temperature magnetic order can be captured within an ex- 
tended Heisenberg-Kitaev model. Our detailed comparison 
of experiment and theory in particular suggests that while 
Na2lr03 is located deep in a magnetically ordered regime, 
Li2lr03 lies close to the spin-liquid regime around the Ki- 
taev limit (a > 0.8). Future experiments will further investi- 
gate whether the application of c-axis pressure can push these 
systems deeper into the orbitally dominated regime, and in 
particular whether Li2lr03 can be pushed into the spin-liquid 
phase. 
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Synthesis and Structure 



Synthesis- Poly crystalline samples of A2lr03 (A = Na, Li) and Li2Sn03 were synthesized by solid state synthesis. High 
purity starting materials A 2 C0 3 (A = Na, Li) (99.995% Alfa Aesar) and Ir metal powder (> 99.95% Alfa Aesar) or Sn0 2 
(99.995% Alfa Aesar) were mixed in the ratio 1.05 : 1 and placed in an alumina crucible with a lid and given heat treatments 
between 750 °C and 950 °C in steps of 50 °C with intermediate grindings and pelletizing after each step. The Li2SnC>3 sample 
was given a further heat treatment at 1000 °C. In an attempt to grow single crystals the Li2lr03 sample was dissolved in excess 
LiCl flux at 850 °C for 6 hrs and then cooled to 590 °C at 3 °C/hr. After washing majority of the LiCl flux with de-ionized 
water we obtained a fine black powder which turned out to be highly ordered Li2ir03 poly crystalline samples. This powder was 
pelletized and sin tered at 900 °C for 48 hrs to get a hard pellet for resistivity and heat capacity measurements. 

Structure .- Powder x-ray diffraction (XRD) scans of poly crystalline A2lr03 (A = Na, Li) and Li2SnC>3 were found to be 
single phase and could be indexed to the monoclinic C2/c (No. 15) structure. Rietveld refinements, [1] of the x-ray patterns 
gave the unit cell parameters a = 5.4187(2) A, b = 9.3689(3) A c = 10.7731(5) A and f3 = 99.598(18) ° for Na 2 Ir0 3 and 
a = 5.1678(4) A, b = 8.9347(7) A c = 9.7825(4) A and (3 = 99.998(14) ° for Li 2 Ir0 3 . A representative XRD scan for 
Li2lr03 obtained by LiCl flux method is shown in Fig.[4]along with the Rietveld refinement shown as the curve through the data. 
Regions where small peaks of LiCl appeared have been excluded in the fits. The inset of Fig. [4] shows the XRD data and Rietveld 
refinement at low angles on an expanded scale. The highly ordered nature of the sample can be seen from the separation of the 
lines between 20 = 19 and 22 ° which in a disordered sample would appear merged together and would also show a highly 
anisotropic shape with a tail extending to the higher angle side of the peak, fid 
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FIG. 4: Rietveld refinement of the X-ray diffraction data of polycrystalline Li 2 Ir03. The closed symbols represent the observed data, the solid 
lines represent the fitted pattern. Regions where small (< 10%) peaks of LiCl appeared have been excluded in the fits. The inset shows an 
expanded view of the data at low angles. 
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Resistivity 

The resistivity p versus temperature T for Li 2 Ir03 is shown in Fig. The data clearly point to insulating behavior with a 
room temperature value w 35 ft cm and an activation gap « 700 K. The insulating resistivity coupled with the local-moment 
magnetism indicates that Li 2 Ir03 is a Mott insulator. 
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FIG. 5: Resistivity p versus temperature T data for poly crystalline Li2lr03 shown on a semi-log plot. 



Details of the Density Functional Theory Calculations 

We applied the WIEN2K [7] implementation of the all electron linearized augmented plane wave method in the LDA+U+SO 
approximation, within the second variational treatment using U=3eV and J=0.6eV. For the lattice structure the empirical mono- 
clinic lattice structure C2/c Jg)] was used. The moments directions were constrained along the three crystallographic axes. The 
moments connected by half a unit cell in the c direction, were aligned antiferromagnetically. The basis size was determined 
by RKmax=8.0 and the Brillouin zone was sampled with a 5x5x2-mesh. For the zig-zag, Neel and stripy configuration, the 
moment direction with the lowest total energy is found to be 'a', 'c' and 'a' respectively. We note that the constrained moment 
directions investigated in this study are not meant to represent the moment direction of the theoretical ground state, which in 
general could be away from the crystallographic axes and non-collinear. We also note that another closely related space group 
C2/m has been suggested for Li2lr03 in the literature |2|]. Since our X-ray diffraction patterns could be fit equally well with 
either space group, we have used C2/c in our DFT calculations for Li2ir03 to be consistent with what was previously used in 
calculations for Na2ir03 120 . 
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FIG. 6: (Up) Total energy per Ir in meV obtained from LDA+U+SO simulations of three antiferromagnetic configurations with the moments 
constrained along the three crystallographic axes. (Down) Lattice and magnetic structure. 



Functional Renormalization Group Calculations 



To capture the experimental data we consider an extended Heisenberg-Kitaev model which is augmented by second and third 
nearest neighbor Heisenberg couplings J 2 and J3, respectively. Explicitly, the Hamiltonian is given by 



#hk-j 2 j 3 = (1 - ol) 1^2 + j 2 Yl + Js Yl \ Si ' °3 

(((i,3)))J 

-2a^2*7a], (2) 

7 

where nearest-neighbor exchange is indicated by the (.) brackets, and second and third nearest neighbor exchange by ((.)) and 
(((.))), respectively. J 2 and J3 are given in units of the nearest neighbor Heisenberg coupling. 

Using a pseudo-fermion functional renormalization group (FRG) calculations we were able to (i) determine the phase diagram, 
(ii) characterize the magnetic ordering of the different phases, and (iii) calculate thermodynamic properties such as the Curie- 
Weiss scale, the ordering scale, and the frustration parameter. Quickly summarized, the main outcome of the FRG calculation 
is the magnetic spin susceptibility as the linear response to a small external magnetic field, calculated via Kubos formula. Note 
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FIG. 7: Phase diagram of the extended Heisenberg-Kitaev model (0 for a = 0.5 with < J2, J3 < 1. Here, the J2-J3 -parameter space is 
scanned in steps of 0.1. The dotted lines illustrate the classical phase boundaries obtained in Ref.0. 



that the magnetic field always points along one of the cubic axes. Since the FRG naturally computes the spin susceptibility 
as a function of a frequency cutoff parameter A, we are able to extract thermodynamic properties of the model system via an 
identification of precisely this frequency cutoff parameter A with a temperature T. Our calculations use cluster sizes of the 
hexagonal lattice of 112 lattice sites. For further technical details about the pseudo-fermion FRG method and the extraction of 
the Curie- Weiss scales and ordering scales from the FRG data we refer to Refs. qI Ifl 

Phase diagram- First, we turn to the overall phase diagram of model (|2). This phase diagram has recently been calculated 
1 4] on a classical level, which revealed four phases with different types of long-range order, i.e., antiferromagnetic Neel order, 
stripy order, spiral order, and zig-zag order (for an illustration of the ordering patterns, see e.g. Ref. |4|). In our approach, the 
inclusion of quantum fluctuations in the FRG flow enables us to treat the system far beyond the classical limit. However, despite 
the quantum nature of our method, we clearly see that in a wide parameter regime the phase boundaries remain roughly the 
same as compared the to classical case. In order to exemplify this result, Fig. [7] shows the phase diagram for a = 0.5 with 
< J2, J3 < 1. This cut through the parameter space exhibits stripy, spiral and zigzag ordered phases with boundaries which 
almost agree with the classical one s. Only in the vicinity of the limits a = or a = 1 quantum fluctuations have a more 
significant effect on the phase diagram and may even destroy the long-range order, see Fig. 3 of the main text. 

Magnetic order- The different types of order in the phase diagram of Fig. [7] also reveal themselves through their different 
peak structures of the spin susceptibility in k- space. Fig. [8] displays these susceptibility profiles in the entire extended (second) 
Brillouin zone for selected parameter points. In the case of stripy order, large response peaks can be seen at an M-point position 
(due to the external magnetic field along one of the cubic axes the susceptibility profile loses its six fold rotation symmetry). 
Upon entering the spiral regime, the ordering peak leaves the commensurate M point and moves continuously towards the Y 
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point. As one approaches the phase boundaries to the zig-zag phase, the ordering peaks broaden in k x direction and finally 
split into two separate peaks which mark the k-space positions of the commensurate zig-zag order. These peaks are located 
in the middle between the T point and the corners of the extended Brillouin zone. Recall that there are also three distinct zig- 
zag ordering configurations corresponding to three differently oriented zig-zag paths in the hexagonal lattice. Each of these 
configurations relates to two ordering peaks at k and — k. As can be seen in the right plot of Fig. [5] the symmetry-breaking 
property of the external magnetic field only allows for two zig-zag ordering configurations, which corresponds to four peaks in 
the Brillouin zone. 

Thermodynamic properties- Finally, we turn to thermodynamic properties of model ©, in particular the suppression of 
magnetic ordering with regard to the Curie- Weiss temperature, which is often considered a measure of frustration and quantified 
by the frustration parameter / = 0/Tn, i.e. the ratio of the Curie- Weiss temperature 6 and the ordering temperature T/y. In 
Fig. |3 we again consider the case a = 0.5. The qualitative behavior of / shown there is characteristic also for smaller and larger 
values of a. The stripy ordered phase can clearly be identified as the region with the smallest frustration, which is in agreement 
with the fact that this order is classically saturated at a = 0.5, J 2 = J3 = 0. As the boundary to the spiral phase is crossed, the 
frustration parameter undergoes a pronounced increase. In general, / grows with increasing J2 and J3 (at least if J3 is not too 
large) and the largest amount of frustration i s obtained near the transition line between spiral order and zig-zag order. Hence, 
by adding J2 and J3 interactions, the frustration can be enhanced by a factor of three as compared to the bare Heisenberg-Kitaev 
case. As shown in Fig. 3 of the main text, / decreases monotonically with increasing a, while the general profile of frustration 
in the J2-J3 plane remains similar to Fig. [9] However, depending on J 2 and J3, for small a a non-magnetic phase may appear, 
which causes a sharp rise of the frustration parameter in the nearby ordered phases. Hence, in regimes of small a, the capability 
of J2 and J3 interactions to increase the frustration is most pronounced. 
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FIG. 9: Total color profile plot of the frustration parameter / = O/Tn for a = 0.5 as a function of J 2 and J3. The dotted lines illustrate the 
classical phase boundaries obtained in Ref.0. 



